BUG: where behaves badly when dtype of self is datetime or timedelta, and dtype of other is not #9804
Conversation
… and dtype of other is not
| values = values.view('i8') | ||
|
|
There was a problem hiding this comment.
I would like to see if we can consolidate some of this logic to com._infer_fill_value (which may need a bit of logic), but I think should work
There was a problem hiding this comment.
I'm not sure I follow. We already know the source and destination dtypes: datetime64 -> int64 or timedelta64 -> float64. What do we need to infer?
There was a problem hiding this comment.
its not that we need to infer, my point is that a very similar code block exists in several places in the code base. needs to be consolidated to avoid repeating is again. Pls look thru and see where it is duplicated (e.g. in internals.py as well)
There was a problem hiding this comment.
I'm sorry, I still don't see the common functionality. You're talking about this function:
def _try_coerce_args(self, values, other):
""" Coerce values and other to dtype 'i8'. NaN and NaT convert to
the smallest i8, and will correctly round-trip to NaT if converted
back in _try_coerce_result. values is always ndarray-like, other
may not be """
values = values.view('i8')
if is_null_datelike_scalar(other):
other = tslib.iNaT
elif isinstance(other, datetime):
other = lib.Timestamp(other).asm8.view('i8')
elif hasattr(other, 'dtype') and com.is_integer_dtype(other):
other = other.view('i8')
else:
other = np.array(other, dtype='i8')
return values, otherand this one?
def _infer_fill_value(val):
"""
infer the fill value for the nan/NaT from the provided scalar/ndarray/list-like
if we are a NaT, return the correct dtyped element to provide proper block construction
"""
if not is_list_like(val):
val = [val]
val = np.array(val,copy=False)
if is_datetimelike(val):
return np.array('NaT',dtype=val.dtype)
elif is_object_dtype(val.dtype):
dtype = lib.infer_dtype(_ensure_object(val))
if dtype in ['datetime','datetime64']:
return np.array('NaT',dtype=_NS_DTYPE)
elif dtype in ['timedelta','timedelta64']:
return np.array('NaT',dtype=_TD_DTYPE)
return np.nanI haven't seen any code block that looks similar to _try_coerce_args elsewhere either. Could you give an example?
|
merged via 1f9b699 thanks! |
There are a few weird behaviors that this fixes:
otheris anintorfloat, thenwherethrows an Exception while trying to callother.viewotheris an np.int64, it works fineotheris an np.float64, there is no error, but the result is bizarre (it reinterprets the bits of the float as an integer, rather than casting it)otheris list-like and has a numerical dtype, then it throws an exception while try to callallon the value False (for some reason, comparing an ndarray with dtype datetime64 and an ndarray with an integer dtype just returns a scalar False rather than a boolean ndarray)